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ABSTRACT 

The current wave of reform in education, often 
referred to as systemic change, requires specific evaluation needs, 
especially at the school district level. Evaluation must address both 
the local district effort for systemic reform and restructuring. The 
state and national need to demonstrate accomplishment of student 
outcomes and adherence to professional standards should be taken into 
consideration. As evaluators of the Michigan Statewide Systemic 
Initiative (MSSI) , a National Science Foundation sponsored program 
for reform of mathematics and science education, the authors operate 
under the premises that everyone is a stakeholder in the evaluation 
and that all stakeholders need to be actively and collaboratively 
involved. A collaborative, constructivist approach to evaluating the 
MSSI gives due consideration to the statewide and local interests* 
The authors used variables related to knowledge accessibility, the 
press for achievement, and professional teaching conditions to 
construct seven instruments for evaluation to allow the district to 
compare itself to similar districts. One table lists the variables. 
(Contains 30 references.) (SLD) 
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ABSTRACT 

The cxurent wave of reform in education, often referred to as systemic change, is 
made up of two philosophically and politically different elements. One, responding 
to years of unsuccessful reform that first focussed on adoption of exemplary 
instructional materials and then attempted to mandate teaching behaviors, calls 
for restructiuing the systems of education to place authority and KJcountability at 
the local level. And do so through processes that involve all concerned in the 
decision making. The other, responding to the frustration of American students 
falling behind in conceptual understanding, skill levels, and ability to apply 
knowledge and skills in comparison to other nations, calls for national standards, 
nationally specified curricula, and increased national standardized testing. 
Evaluation plays a key role for both these elements. For the latter it is a well 
established role based on years of research in testing and measvirement. For the 
former, new theoretical bases have been developed, practices to support this work 
have begun to emerge and evaluators are struggling with new roles. 

This paper suggests the natvire of the specific evaluation needs in a world of 
systemic change, especially at the school district level, and describes work in 
progress which seeks to meet those needs. The authors are co-evaluators of the 
Michigan Statewide Systemic Initiative, a National Science Foundation sponsored 
program for reform of mathematics and science education. Evaluation strategies 
described serve both local districts engaged in restructuring and state and 
national program personnel and policymakers. 
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The Role of Evaluation in Systemic Change in Education 

There must be a fundamental change in the way that education research 
and evaluation are perceived and conducted . . The era of the objective 
observer of education reform, standing off at a distance and evaluating what 
goes on is a completely passe notion. Evaluation, observation, and research 
has to be an integral part of and directly involved in the change process, 
because otherwise it means nothing. 

Anonymous, Science Education for the 1990's, StJohn, 
1992, p.48 

The history American education in the 20th Century and in pai-ticular the last 30 
years is told differently by different writers seeking to support political positions 
or newly proposed program directions. Nonetheless, almost without exception the 
word being used to describe the most recent efforts for improvement is 
restructuring (Darling-Hammond, 1993). Some date serious efforts to change 
American education to the publishing of A Nation at Risk (Bell, 1993), others 
describe at least 30 years of school reform and seek in restructuring a "new" wave 
that holds the most promise for success (Sashkin & Egermeier, 1992). Fullan 
(1991) describes the most recent decade as a tug of war between two opposing 
reform approaches, intensification and restructuring. According to Fullan, both 
are comprehensive, intending systematic change; but they are philosophically and 
politically at odds. He defines intensification as including increased definition of 
curriculum, mandated textbooks, standardized tests aligned with cvirriculum, and 
specified educational methods backed up by evaluation and monitoring. 

In contrast, restructuring involves a release of tight controls to permit local 
decision making through such forms as site-based management, increased 
professionalization of teachers, restructured schedules and timetables, shared 
values and goals, and new roles for students, teachers, administrators, and 
policymakers. Darling-Hammond (1993) posits a similar dichotomy calling the two 
competing models the Theory X type based on standardization and tightened 
controls and the other, the new paradigm of school reform with roots in 
progressivism, seeking to develop communities of learning. She speaks out 
strongly for new policies that support schools in the process of developing 
democratic discourse as they move to become such communities of shared purpose. 



Neither of these two competing movements is going to go away in the near future. 
And in fact, real systemic change in education requires the maintenance of a 
tension between top down mandated standards with clearly defined outcomes as 
goals, and locally derived goals and objectives with the ways and means to achieve 
standards and produce the outcomes all childrerx deserve. Evaluation has played, 
and continues to play a significant role in the intensification movement. The art 



The Role of Evaluation in Systemic Chanje in Education - Barley, Z.A. & Jenness, M. Page 4 



and science of large scale measurement is well developed. This is not true for 
evaluation at the local level in an era of restructuring. A differetxt theoretical and 
philosophical stance is required of evaluation and new capabilities must be 
developed in evaluators and in the users of evaluation reports. 

The term systemic change is a priori applied across levels of the educational 
'system' and hence readily encompasses both sides of the dichotomy. This can 
make it a more useful term to apply to educational change today than 
restructuring. It is important, however, that the need for serious restructiiring be 
understood and implemented at all levels if meaningful change is to occur in 
American education. Sashkin and Egermeier (1992) offer this definition: 

Restructuring involves changes in roles, rules, and relationships between 
and among students and teachers, teachers and administrators, and 
administrators at various levels from the school building to the district 
office to the State level, all with the aim of improving student outcomes, (p. 
12) 

The missing element are the policymakers. When their voice is included and it is 
tmderstood that their roles, rules, and relationships must also be restructured, the 
stage is set for meaningful change. 

What Have We leameH From Faile d Reform Efforts ? There is no question among 
policymakers, researchers, and educators that the efforts of the past have failed. 
Roemer (1991) writes poignantly of a collaborative effort at school reform between 
secondary school and university faculty that failed. Kirst (1993) discusses the 
weaknesses of the American education system on the other side of a decade of 
reform. Sashkin and Egermeier (1992) systematically document three waves of 
reform (fix the parts, fix the people, and fix the school) that can improve education 
but "change based on either one or some combination of these strategies is 
incremental, is often temporary, and is sometimes totally absent - or even for the 
worse" (p. 16). 

Fortimately we have learned much from the experiences of decades of reform that 
can shape new successful strategies for systemic change. The Wingspread 
Conference (St.John, 1992) on strategies for systemic change in science education 
developed a list of 27 characteristics to contrast a project-based change model with 
a systemic change approach. The authors condensed the list into the following 5 
systemic approaches: (a) build the infrastructure for reform, replacing system 
elements rather than adding new ones, (b) build on system strengths rather than 
fixing deficits, (c) understand that refoi-m vr, a long term evolving process, (d) focus 
change on oneself first rather than on othr.rs, building capacities and marshalling 
resources and (e) place power in the hands of those in the system. Several writers 
emphasize the need to change the culture of the school to support systemic change 



t 



The Role of Evaluation in Systemic Change in Education - Barley, Z.A. & Jenness, M. Page 5 



(Darling-Hammond, 1993; Fullan, 1991; Sarason, S.B., 1990; Sashkin & 
Egermeier, 1992). Thus two themes predominate, involve everyone who has a 
concern in the change process, and provide them the capacity, authority, and 
methods to succeed. 

At a more detailed level, Fullan (1991) offers a careful analysis of how change is 
initiated and then carried out. He identifies eight factors associated with 
initiation: (a) The existence and quality of innovations, (b) access to such 
innovations, (c) advocacy from central administration, (d) teacher advocacy, (e) the 
involvement of external change agents or facilitators, (f) community 
pressure/support/apathy, (g) new policies and funds from the state or national 
level, and (h) the capacity of the district to problem-solve and not be hamstrung by 
biireaucracy. 

Newman (1993) sums up the lessons of past reform concluding we know what to 
do, and asks, why haven't we put into practice what we know. From an informal 
poll of state and national policymakers he identified five barriers to changing the 
system: (a) there is not a critical mass of oeople who understand what systemic 
change is, (b) there is no xinified, turfless or widely shared vision for education in 
a state, (c) the policy making process is fragmented, (d) systemic change is 
complex, enormous, and politically difficult to manage, and (e) the public doesn't 
see the need for change. The dcmain for evaluation of systemic change efforts 
therefore includes the approaches used, the culture in which it occurs, the 
understanding of the array of factors brought to bear and a knowledge of barriers. 

Rrnarl ARRnmpt.ions for Evaluat ion in a Systemic Change Process Drawing from 
these lessons learned and the perceived barriers, the authors in their work with 
the evaluation of the Michigan Statewide Systemic Initiative (MSSI), have worked 
under several broad assumptions. The evaluation must address both dimensions 
of systemic change, the local district effort for systemic reform or restructuring 
and the state and national need to demonstrate accomplishment of student 
outcomes and adherence to professional standards for mathematics and science 
education. Each is dependent on the other for direction and on the other's full 
understanding of the systemic change process for success. Therefore both must be 
addressed in concert. More critically, the evaluation must itself model and 
therefore support the strategies for successful systemic change learned from the 
preceding decades. As MSSI evaluators we work under these premises: 1) 
everyone is a stakeholder in the evaluation, parents, students, teachers, 
administratcrs, the business community, policymakers, government agencies, the 
general public; 2) stakeholders need to be actively and collaboratively mvolved in 
the evaluation; 3) there must be a shared vision and purpose undergirding the 
evaluation; 4) the evaluation serves two masters, accountability and the 
plannin^strategizing process to carry out systemic change; 5) evaluative 
information must be user-friendly and available promptly for decision making; 
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and, 6) all of those involved in the evaluation must build new capacities to conduct 
and use evaluation in a systemic change effort. 

The section that follows draws upon current work in evaluation describing several 
more recently developed approaches which may serve as resoiirces for systemic 
change evaluation. 

Emergent Approaches to Evaluation Applicable to Systemic Chan ge Evaluation 
Joseph S. Wholey writes to evaluators from a long and distinguished career in 
evaluation in federal programming. His primary concern is the use, or lack of use, 
of evaluation information by program managers to improve programming. With 
his focus on government social programs and his belief in their mandate to be 
responsive to public needs, his voice is particularly relevant to reform in 
education. He notes that government programs, like educational systems, are 
rarely terminated and therefore all the more must be constantly reviewed for 
improvement. Thus for Wholey, program evaluation must be directed toward 
program management needs. He defines it accordingly: "Program evaluation is 
the measurement of program performance, the making of comparisons based on 
those measurements, and the use of the resulting information in policy-making 
and program management" (1979,p.l). From this position he moves toward 
identifying why program managers were not using program evaluations and how 
evaluation might change to increase use. One direction he suggests is in the role 
of evaluators: 

The new evaluator role is a program advocate - not an advocate in the sense 
of an ideologue willing to manipulate data and to alter findings to seciire 
next year's funding. The new evaluator is someone who believes in and is 
interested in helping programs and organizations succeed. At times the 
program advocate evaluator will play the traditional critic role: challenging 
basic program assumptions, reporting lackluster performance, or identifying 
inefficiencies. The difference, however, is that criticism is not the end of 
performance-oriented evaluation; rather it is part of a larger process of 
program and organizational improvement, a process that receives as much 
of the evaluator's attention and talent as the criticism function. (Bellavita, 
Wholey, & Abramson, 1986, p.289). 

In his 1979 article Wholey also describes evaluatior. tools which he believes can 
assist in carrying out such a role: evaluability assessment, rapid feedback 
evaluation, performance monitoring, and intensive evaluation. 

Lee J. Cronbach and associates in their 1980 book Toward Reform of Program 
Evaluation called for a transformation of evaluation in order to meet its mission 
"to facilitate a democratic, plviralistic process by enlightening all the participants" 
(p, I), With this mission in mind, he strongly criticizes usual assumptions about 
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appropriate practices in evaiuation and suggests that "an evaluative study of a 
social program is justified to the extent that it facilitates the work of the polity. It 
is therefore to be judged primarily by its contribution to public thinking" (1980, p. 
64). His contribution to the evaluation of systemic change in education is 
therefore, not unlike Wholey^s, a refocussing of energies on serving the public 
good. He has much to teach the evaluator about the stages of program 
development and the role of evaluation at different stages. One model that may 
be of particular help to systemic change evaluation is the concept of a Social 
Problem Study Group. The group would be made up of members representing all 
concerned parties and would study problems in the broadest possible way. They 
would keep themselves informed by listening to the entire array of stakeholders 
and would develop a comprehensive interpretation of the progress of the reform, 
not tied to ary one source. The Social Problem Study Group would constantly 
reformulate key questions for the evaluation to address and keep the work in the 
proper time perspective for all audiences (Cronbach et.al., 1980). 

Other more recent writers have addressed the involvement of stakeholders in the 
evaluation itself. Fetterman (1993a) in his book. Speaking the Language of 
Power: Communication, Collaboration, and Advocacy, develops the concept of 
empowerment evaluation. In an unpublished paper addressed to the members of 
the American Evaluation Association he says, "Empowerment evaluation is the 
use of evaluation concepts and techniques to foster self-determination. The focus is 
on helping people help themselves. This evaluation approach is problem focussed, 
collaborative, and requires both qualitative and quantitative methodologies" 
(Fetterman, 1993b). 

Guba and Lincoln have fully developed what they designate as a new generation 
in evaluation in their 1989 book. Fourth Generation Evaluation. They 
characterize it as responsive constructivist evaluation, responsive in that it 
determines parameters and boimdaries through an interactive, negotiated process; 
constructivist in methodology that rejects the scientific experimental approach in 
favor of creating a constructed reality that incorporates the observer/observed 
interaction. The processes of fourth generation evaluation entail identifying all 
the relevant stakeholders, and eliciting from them the claims, concerns, and issues 
they wish to raise. Then the evaluation provides a context and a methodology 
through which these persons and their concerns are part of a negotiated 
evaluation process up to and including generated reports. Here, too, the role of 
the evaluator requires different skills than usually though necessary. 

Donmeyer (1991) also draws his theoretical underpinnings from the postpositivist 
paradigm. His emphasis is on the political nature of knowledge. He 
reconceptualizes evaluation, basing it on six assumptions: educational issues are 
ultimately more conceptual than empirical, questions of meaning should be 
addressed as part of the evaluation, meanings are not true or false but must be 
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resolved through discourse, if questions of meaning are resolved the 'truth' may be 
discernible, and the evaluation should produce not only recommendations for 
change, but also expand the understanding of the participants in the evaluation. 
He translates thes(3 assiomptions into a six-stage evaluation process including the 
formation of a representative forum and a consensus process for resolving issues. 

These are just a few examples from an emerging literature supporting the 
philosophical precepts of successful systemic change as we now understand it. 
Practices to support the evaluation of systemic change efforts are also evolving, 
several currently in use are described below. In all cases, the role of the evaluator 
is expanding and changing as is the role of the evaluation itself. 

Work in Progress in Svstemic Change E valuation: the M SSI Experience The 
Michigan Statewide Systemic Initiative in mathematics and science education, 
funded by the National Science Foimdation as part of a national reform program, 
was created to reform education using a systemic change approach. A shared 
state\vide vision for mathematics and science learning and teaching for all 
students and teachers underlies the effort. The State akeady had developed core 
curricula and essential goals and objectives. Emphases are on improving 
outcomes for all students with special attention given to those underrepresented in 
mathematics and the sciences and on those who live in extreme urban and rural 
areas perceived to have less access to resources to support mathematics and 
science education. Components of the initiative include an examination of existing 
policies and programs at the state and local level, a coordinated approach to 
reforming teacher education, connecting the wide variety of other initiatives in 
mathematics and science education in the state for dissemination and professional 
development, and the 3election of a group of local districts to become model 
districts in undergoing systemic change efforts. The model districts are expected 
to develop community coalitions to support reform and create action plans that are 
based on a systemic change philosophy. The evaluators serve as members of the 
MSSI Management Team. 

The authors proposed a collaborative, constructivist approach to the evaluation 
from the beginning, with an emergent design flexible enough to meet the changing 
needs of the MSSI program. Such an approach was well received locally and in 
Washington. Foxir broad evaluation questions form the core of the evaluation. 
Two are concerned with long term student and teacher outcomes, two are focussed 
on the success of the systemic change process. In the material that follows several 
strategies of the evaluation are desQibed that are intended to accomplish the work 
under the premises listed earlier in the chapter. 

Strategies ior.S^^^ Four primary strategies 

strengthen the evaluation at the state level: internalization of the evaluation in 
the strategic plans for each component., the creation of The Evaluation Collegju m. 
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the use of a inetaevaluation panel, and the restructuring of the state database for 
educational inforraation. 

The evaluators facilitated a strategic planning p rocess for the components as a 
prelude to and a basis for the development of the evaluation plan. Each 
component director and staff reviewed and revised the long term goals from the 
proposal and developed end of project objectives for the component. The entire 
MSSI Management Team then re\ne A the group of goals and objectives. The 
evaluators developed a list of evaluation questions for each component to match 
the goals and strategies proposed. These questions examine three aspects of the 
work: the intended outcomes and whether they are obtained, the context in which 
the work is carried out, and the actualization of the planned implementation. The 
latter two types of questions are essential to understanding systemic change - 
outcomes alone will neither provide useful information to program staff nor make 
data available about the various sources which may influence an outcome. Once 
the expected outcomes and the evaluation questions had been negotiated the 
component directors developed plans for the year with timelines and 
responsibilities. These were also revievv^ed for feasibility and the systemic 
characteristics of the plan. Next the evaluators developed the evaluation plan for 
each component by identifying process and outcome indicators for both the one 
year plan and for the five year goals and objectives. These were also negotiated 
and data collection responsibilities assigned to either evaluators or component 
staff. This has proved to be an efficient and effective method for collaboratively 
involving the component staff. Additional stakeholders are reached through 
individual evaluations. School district personnel who attended a first technical 
assistance session, for example, completed a survey instrument. The data were 
compiled within two days and returned to the project staff for modification of the 
next session. At the next session, the district personnel v/ere provided with a 
summary of their evaluation feedback and asked for further comments. Periodic 
interviews are used for stakeholders who have a more sustained involvement with 
the work. Again, the information gained is intended as much, if not more, for 
shaping the work in progress than for sximmative pxirposes. 

Even with the careful inclusion of stakeholders in the strategic planning process 
and the collection of data from a variety of involved persons, there was concern ^ 
that a large number of important stakeholders might be overlooked. With this in 
mind the evaluators formed T he MS SI Ev aluati on jQollegjum, an informally 
organized statewide open membership 'group' who agreed to be the eyes and ears 
of the evaluation as they attended to their usual work in science and mathematics 
education. Members of several existing organizations were encouraged to 
participate and a membership list created. These members receiv^^ the MSSI 
evaluation newsletter, Valuing, and are periodically asked in the newsletter to 
respond to brief questionnaires or to survey their constituents or local districts 
about recent initiatives. FVesentations at statewide educator conferences are also 
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used to engage stakeholders in the evaluation and obtain their feedback. 

A carefully selected metaevaluation pan el has been created, made up of persons 
within and outside of Michigan involved in systemic reform related to 
mathemdtics and science education. Whereas the usual emphasis of such a panel 
might be on the adherence to standards for exemplary evaluations, the concern for 
this panel is whether the evaluation is capturing the complex variables needed for 
systemic change evaluation and whether it embodies systemic change thinking in 
its plans and procedures. 

Finally, a key sti^ategy originally intended to be especially responsive to the 
resialtg/outcomes oriented stakeholders, the MSSI dat abase, has already 
demonstrated the importance of information to the entire systemic change process. 
The Michigan Department of Education has been extremely cooperative in 
providing access to existing data. Tbte condition of the available data is a dramatic 
indication of why previous attempts at reform have met with limited success. 
Data have never been collected that are defined and organized to serve the needs 
of program improvement. Data that are collected are strictly accountability data 
and the requesting agents are not program managers but external report writers 
simply wishing to plug a number into a report field. These external persons have 
tended to be in charge of the definitions and frequency of data collection. Local 
districts or state program persons seldom see the use of the data to correct 
misperceptions nor do they have access to findings. For example, the mmiber of 
middle school teachers teaching mathematics was recently grossly vmderreported 
for the State because of such problems and only picked up when the national 
report came out and the problem was revealed. In addition to directing data 
collection and analysis to program improvement, systemic change requires entire 
new areas of data collection, e.g. the natiire of actual classroom teaching, the 
preparation of teachers in new modes of teaching, decision making processes in 
use, school improvement plans, state and local policies and procedures that 
provide for or restrict change, and facilities and materials and their accessibility. 
For these reasons the evaluation includes the development of a database, involving 
far reaching decisions on what is collected and how, guided by systemic change 
needs. Such a database itself becomes a part of the change process and the 
infrastructui'e for continuing change. 

It is inevitable that additional strategies for evaluation that measure and support 
statewide change will emerge and "ome of those now in use will fail and be 
discarded. Those that are necessary and effective in Michigan may not be 
relevant in other settings. We offer them only as directions in which to look in 
forming systemic change evaluations at the state level. 

Strate_giesJ!pxLoc.aLDi.stilct^%^^ As part of the MSSI 

effort, a group of local school districts has been selected from among those that 
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submitted proposal^action plans to participate in a foiir year process of systemic 
change, including an emphasis on building community coalitions to support 
reform. The selected model districts will not only carry out their independent 
plans but they will also be expected to operate as a group. The desip for the 
work of the districts as a group has been drawn from the evaluators' work with 
cluster evaluation (Barley & Jenness, 1993a, 1993b; Jenness & Barley, 1992a; 
Jenness & Barley, 1992b). This strategy is discussed more fully in the material 
that follows. 

Four strategies for evaluation at the local level are currently underway, others 
will certainly be required; two, evaluation of contextua l indicators and focus 
groups on change , apply to specific issues, and two, as briefly mentioned above, 
a ction plans and cluster evaluation , are broad strategies. 

As mentioned in the section on the database, an area of importance in developing 
action plans for districts is the local context. The evaluators have developed two 
approaches to assessing local context, one assesses the context to support student 
achievement and the other the context to support change itself. The f?rst is an 
instrument to assess a set of contextual ind icators . Jeanne Oakes (1989) reviewed 
the literature on student achievement in math and science "to identify policy- 
relevant school-level indicators . . . (that) promise to be useful for monitoring and 
improvement" (p. 182). She makes the case: "Context indicators can be used to 
measure schooling resources and processes; they may help forestall educators' 
tendency to narrow their programs in order to "look good" on limited outcome 
measures; and they can provide information about the context in which particular 
outcomes are achieved' Op- 181). Oakes (1989) organizes the findings from .he 
literature into three composite school-level indicators: access to knowledge, press 
for achievement, and professional conditions for teaching. Under each of these 
concepts are a list of variables that together can torm indices for the indicators. 
See Table 1. 
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Table 1 Vflriahlfis Relate d to S nhnnl Context Indicators 
ACCESS TO KNOWLEDGE 

Teacher QualificatioriS 

Instructional T'me 

Course Offerings 

Class Grouping Practices 

Materials, Laboratories, Equipment 

Academic Support Programs 

Enrichment Activities 

Parent Involvement 

Staff Development 

Faculty Beliefs 

PRESS FOR ACHIEVEMENT 

Focus on Academics 
Graduation Requirements 
Graduation Rates 
Enrollment in Rigorous Programs 
Recognition of Academic Accomplishments 
Academic Expectations for Students 
Uninterrupted Class Instruction 
Administrative Involvement in Academics 
Quantity and Type of Homework 
Teacher Evaluation Emphasizing Learning 

PROFESSIONAL TEACHING CONDITIONS 

Teacher Salaries 
Pupil Load^Class Size 
Teacher Time for Planning 
Collegial Work 

Teacher Involvement in Decision-making 
Teacher Certainty 
Teacher Autonomy/Flexibility 
Administrative Support for Innovation 
Clerical Support 

Using these variables the evaluators constructed a set of seven instruments, a 
district superintendent survey, principal surveys for each level (elementary, 
middle, and secondary), and three teacher surveys according to level. Results 
from the surveys allow the district to compare itself to similar districts, to 
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compare teacher responses with principals, and to assess areas of contextual 
strengths and weaknesses for consideration in action planning. For the MSSI 
Management Team the results present a picture of Michigan school context across 
a variety of district types as a tool in planning inservices, focusing RBPs, awardin.g 
federal pass through funds, and designing technical assistances for the model 
districts. 

Many authors Avriting on systemic change in local school districts speak to 
problems connected with organizational readiness (Dellar, 1993; Fullan, 1991; 
Jenlink, 1993). As Dellar indicates, "while the impetus and direction for reform 
might stem from forces external to the school, it is the school-level personnel that 
require both the preparedness and capacity to implement change" (1993, p.2). In 
his work, Dellar has used two questionnaires in addition to interviews and other 
data sotirces. The School Organisational Climate Questionnaire (Dellar, 1990) 
gathers information about decision-making, innovation and change. The second 
questionnaire is administrated to staff members and contains information on 
principal leadership style, administrative decision-making, sub-system linkage, 
and attitudes towards restructuring (Dellar, 1993). From his findings, Dellar 
recommends assessing school organizational climate to determine readiness and, 
where the climate is poor, work toward improving the climate before undertaking 
major restructm-ing efforts. He also suggests that if sub-system linkages ai'e weak 
because of the dominance of, for example, departmental allegiances, supporting 
and strengthening linkages should be part of the plan. Finally, he recommends 
strategies to develop for leaders knowledge and skills for multi-directional 
communication and authentic participatory decision-making, for collaboration such 
as consensus building, and for decision-making such as strategic planning, priority 
setting or evaluation. 

Jenlink (1993) describes the use in a 2500 pupil, eight building school district of a 
focus group i nterview method based on Kjrueger's document on focus groups 
(1990) to assess readiness for organizational change. Jenlink conducted 24 focus 
groups within the district including groups of students, teachers, administrators, 
auxiliary personnel and citizens. Group interviews focussed on the need for 
change, identification of stakeholders for district change, kinds of changes, and the 
district's track record for change. The authors have planned the use of focus 
groups in the MSSI model districts as an assessment tool for use by the model's 
leadership and as a measure of context as a contributing factor in relative success 
or failure of the reform efforts. 

The nature of the development of an MtipiLplan_or. strMe^cjplan for systemic 
change in local school districts varies dependent on the district's history of 
planning. In some districts a school improvement plan may exist that in itself 
suggests excellent direction but lacks a systemic approach. In other districts a 
failed strategic planning effort may be a stumbling block to all planning efforts for 
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a period of time. Some districts have a variety of specific plans with no 
connections between them. How the action plan for systemic change is shaped 
depends to a large extent on this backdrop. A need for capacity building for 
planning is another important factor. There are good resources available both for 
a simple but effective how-to (Barry, 1988) and for more extensive discussions of 
the issues (e.g. Steiner, 1979). The value of the planning process lies in the 
involvement of stakeholders, in learning collaborative, consensual working 
processes, in de^/'eloping ownership of the systemic change process, in allocating 
workload and setting timelines, in planning for monitoring of success and failure 
and of implementation, and in determining the need for new resources and 
training. 

Finally, a cl uster eva l uation approach has proven to have value at the district 
level and may be useful for smaller change units within a large district. The 
authors have described cluster evaluation extensively elsewhere (Barley & 
Jenness, 1993). A cluster is a group of districts, or other entities, setting out to 
accomplish similar purposes within somewhat similar settings. The authors have 
worked extensively with two clusters of 12 districts or entities funded by the 
W.K .Kellogg Foundation to improve science education, a voluntary network 
(cluster) of 22 independent mathematics and science centers, and are now working 
with the group of 25 MSSI model districts. The cluster entails bringing districts 
together up front to define common objectives, agree upon common data collection 
methods and instruments, better prepare staff in evaluation, and provide mutual 
strengthening through regular networking conferences. The conferences serve as 
important occasions to build capacities, master new content, share lessons learned, 
joint problem solve, and share resources. In an effort such as systemic change for 
which new ways of thinking and doing business must be mastered, the help that 
can be obtained through a coxxmion effort is essential. Administrators find 
colleagues in learning new roles, teachers find other models for relating and 
teaching, community members find ways to tap coxxmiunity resources and change 
community attitudes. As a group, the cluster has more power to influence 
policymakers or leverage resources. For the evaluators, the common elements of 
the change process and common monitoring methods, contrasted with different 
local contexts and implementation strategies presents a rich environment to learn 
about systemic change. 

Conclusions The role of evaluation in systemic change can and should be central. 
The success of systemic change initiatives is dependent on information both in 
order to make critical decisions about the work and to continuously involve 
stakeholders in the ongoing process. Evaluators can not sit back and hold on to 
information nor have the luxury of developing it on a timeframe that suits 
sumimative purposes but denies immediate program management needs. 
Evaluators musty be advocates when their data show a need to move in a 
particular direction. 
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Because of the collaborative nature of systemic change, the centrality of 
information, and the need for capacity building in evaluation; evaluators become 
educators in the knowledges and skills that make for good systemic change 
evaluation. To engage all key players means granting them not only the capacity 
but also the authority - power - to change the course of events. Evaluators must 
then become negotiators attempting to persuade but willing to yield, in arriving at 
an evaluative process that serves the program effectively. Fundamental changes 
are not easy but trends in the discipline of evaluation have provided insights and 
directions. Methods have begun to emerge to support such efforts. And for 
evaluators to see their work become part of sweeping changes in education rather 
than gather dust on shelves is reward indeed. 
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